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There is growing interest in a phenomenon called the "synfire chain" , in which firings of neu- 
rons propagate from pool to pool in the chain. The mechanism of the synfire chain has been 
analyzed by many resarchers. Keeping the synfire chain phenomenon in mind, we investigate 
a layered associative memory neural network model, in which patterns are embedded in con- 
nections between neurons. In this model, we also include uniform noise in connections, which 
induces common input in the next layer. Such common input in layers generate correlated fir- 
ings of neurons. We theoretically obtain the evolution of retrieval states in the case of infinite 
pattern loading. We find that the overlap between patterns and neuronal states is not given 
as a deterministic quantity, but is described by a probability distribution defined over the em- 
semble of synaptic matrices. Our simulation results are in excellent agreement with theoretical 
calculations. 
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§1. Introduction 

Abeles proposed a synfire chain as a model of the cortex. ^ It consists of neuron pools connected 
to each other in a feedforward chain so that firings can propagate. Repeated spatiotemporal firing 
patterns with precise timing accuracy within milliseconds have been recorded in the frontal cortex 
of monkeys performing tasks. 2 ) These findings suggest the occurence of some temporal structure in 
the brain. In recent experiments, temporally precise firing sequences of spikes have been reported 
for a songbird vocal control system. 3, 4 ) It was also proved that sequences of synchronous activity 
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propagate through a reproduced multilayer feedforward network of neurons in an in vitro slice 
preparation of rat cortex. 5 ~ 7 ) 

Models of a synfire chain and its mechanism have been studied theoretically by many re- 
searchers. 8 ~ 17 ) Recently, Diesmann and Cateau et al. investigated conditions for stable propagation 
of correlated firings along the feedforward networks. 11 ' 12 ) 

Amari et al. analyzed a model of a neuron pool in which neurons receive common overlapping 
input, and proved that higher order interactions of neurons exist in synchronous firing, which 
generates wide-spread activity distributions. That is, the self-averaging property breaks down even 
in the stochastic limit, and activity depends on the sample, therefore activity distribution is not 
concentrated on its mean value and has a wide-spread distribution. However, in their model, 
neurons are uniformly connected to each other and each connection has no structures such as 
learning efficacies. 

In the present study, we investigate a layered associative memory neural network model in which 
connections patterns are embedded together with uniform noise which induces common input into 
the next layer. Common input in layers generates correlated firing of neurons. We discuss cases of 
infinite pattern loading in the networks. 

§2. Layered Associative Memory Model 

We consider a layered associative memory model. In this model, neuron states are denoted by x\ 
(neuron at i site in layer £), which takes +1 or —1 corresponding to the firing state and nonfiring 
state, respectively. Firing of neurons is propagated according to 

^ +1 = ^E4*s). (!) 

where the layer number is i = 1, ■ • ■ , L and the number of neurons in a layer is i = 1, ■ ■ ■ , N. F is 
the output function F{h) = sgn(/i). For the coupling of neurons, we consider the following form: 

i p 

ij,=i 

Here, the first term describes correlated learning in which the patterns are embedded. Each com- 
ponent of the patterns is assumed to be an independent random variable that takes a value of either 
+ 1 or —1 according to the probability, 

Prob[£f" = ±1] = i ( M =l, £ = 0,---,L), (3) 

where the number of patterns is p = aN . The second term of Eq. (2) is uniform Gaussian noise 
Wj ~ A/"(0, 5 2 /N). The coupling Wj propagates a firing of xj to the next layer uniformly, so it 
induces synfiring in the next layer. 
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We introduce the order parameter characterizing memory retrieval, namely, the overlap with the 
/ith pattern in the £th layer, defined by 



N 



m 



AT Z^^i X «- 



i=l 



(4) 



§3. Theory 

3. 1 Order parameter equations 

In the following analysis, we consider the situation where the pattern jjl = 1 is being retrieved, 
i.e., rrr/' 1 = rr/ ~ 0(1), and m?'^ 1 ~ 0(1/ y/N). In this case, the input to the £ + 1th layer can be 
written as 

hl^4A = ^ + 4 + ri\ (5) 



where 



4^EE^ +1 ^4 

3 M>2 



£ £ 



(6) 
(7) 



The first term of Eq. (5) is a signal term, the second term is the cross-talk noise term, and the last 
term is the uniform noise term rf ~ A/"(0, 5 2 ), which is independent of i. 
The evolution of the neuron states then is given by 



x^ = F(h i ) = F(e i m e + 4 + V e ). 



(8) 



We note that in Eq. (8), rf acts as a uniform threshold, so it induces synfiring in the I + 1th layer. 

Let us now derive equations describing the evolution of the order parameter rr/. We can then 
follow the standard procedure 18 ) to expand Eq. (6) as 
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We then obtain 
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where we have introduced the following two quantities: 



jje+i = ±y F ' 
N 4^ 

3 



k 



(11) 

(12) 



As usual, we assume that z\ obeys a Gaussian distribution, where the mean equals and the 
variance of zf, E[(zf +1 ) 2 ] = (cr e ) 2 , is given by 



T e+i\2/ TT e\2 



+i\2f TT e\2/ TT e-i\2 
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a + (U t+1 Y{a l f. 



(13) 



Finally, when m e , o l and if are given, we obtain the following coupled order parameter equations 
using F(h) = sgn(h): 



m 



' n (m e , a e , if) = J D/(e +1 F^ +1 m e + z l + r,% 



= - [erf (it) + erf(v)] , 



U e+1 (m e , a e , if) = J Dz e (F'(t e+1 m e + / + rf)) 6 



2ira e ^ 



+ e 



(14) 



(15) 



n I dz e ( (z e f 

Dz~ = , „ exp - 



2(^) 2 ) ' 

where u = (m e + ij e ) / (y/2a e ) , v = (m e — rj e )/(^/2a e ) and erf(x) = ^= Jq e~ y2 dy. ( )g denotes the 
average over random patterns of one layer. Here, we use sgn'(x) = 2S(x). 

3.2 Probability distribution function 

Let us now discuss the effect of n theoretically. In the presence of rf, the evolution of x\ is given 
by Eq. (8). For a given if, the order parameters at the £ + 1th layer are given in terms of those at 
the Ith layer as Eq.(13)-(15) However, if is distributed in each layer £ according to the Gaussian 
distribution, and thus the evolution is described by the distribution function p(m e ,a e ). In general, 
the distribution is described by the three parameters as p(m e , a e ,rf). In this case, they can be 
factorized as p(m e ,a e )p(ri e ) and thus the evolution of the distribution p(m e ,a e ) can be described 
by 

(16) 



p{m i+ \o- i+1 ) = I dm e J da 1 J drf l p(m e , o*)p(rf) 
x5(m e+1 - m e+ \m e , a e , r l e ))S(o J+1 - a e+ \m e , a*, rf)), 
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where S(-) is Dirac's delta function and p(rf) = ^^ sl exp ^— 2(V) 2 ) ' 
If we set 

rr/ +1 (mVW) = f(m e ,a e , V e ) = ^ [erf (u) + erf (v)} , (17) 

= g(m e y,V e ) = y« + ^(e- M2 +e- 2 ), (18) 
transforming m , cr^ into f(m, cr , rf),g(m e , a , rf), we get the evoution equation of the distribution, 



a/ 


dg 


_ j9SL 


Of 




dm 1 




dm e 
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m e =m e * ,cr e =a e * 



where m e *,a e * satisfy eqs.(17) and (18). Then we obtain the probability distribution function as 



where u* = (m e * + rf) / '(y/2a e *) and v* = (m e * - rf) / \\f2a 1 *) . 
§4. Results 

We show the simulation results of Eq. (1) to see the effects of 77. As an initial condition, we set 
the state of the first layer according to the following probability: 

Prob[x° = ±1] = j ^ . (21) 

Figure 1 shows the evolution of the overlap m in the case of no uniform Gaussian noise (5 = 0). 
In this case, the evolution is deterministic. Figure 2 shows the simulation results including noise 
7]. In this case, the evolution is not deterministic, but distributed. In Figures 3 - 6, we plot the 
distribution function p(m) for the four layers (£ = 10, 20, 30, 100), where p(m) = J dap(m, a). The 
histogram shows simulation results (N = 10000,1000 samples). The solid lines are theoretical 
calculations of the distribution p(m). We find excellent agreement between simulation results and 
calculations. According to the evolution of layers, one can see that the distribution is peaked at 
two states, namely, the nonretrieval state and the retrieval state. It is noted that the nonretrieval 
state does not exist in the finite loading case. 

Figure 7 shows the distribution function p(m,a) (a = 0.2,5 = 0.2, m° = 0.45, t = 100). One 
peak with m close to zero corresponds to the nonretrieval state. The other peak with m close to 
one corresponds to the retrieval state. Integrating p(m,a) over m, we obtain p(a). Two peaks 
corresponding to the retrieval state and the nonretrieval state can be seen in Fig. 8. 

§5. Summary 

We investigated the layered associative memory neural network model, in which patterns are 
embedded in connections between neurons. In this model, we also include uniform noise in connec- 
tions, which induces correlated firings of neurons. We theoretically obtain the evolution of retrieval 
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states in the case of infinite pattern loading. We find that the overlap between patterns and neu- 
ronal states is not given as a deterministic quantity, but is described by a probability distribution. 
Our simulations results are in excellent agreement with theoretical calculations. 
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Fig. 1. Simulation results for the evolution of m for a = 0.2, 8 = 0, N = 10000, m° = 0.45. We plot the results for 
20 samples. 
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Fig. 2. Simulation results for the evolution of m l for a = 0.2, S = 0.2, N = 10000, m° = 0.45. We plot the results 
for 20 samples. 
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Fig. 3. Layer=10. The histogram is simulation results (N — 10000, lOOOsamples). The line is a theoretical calcula- 
tion of the distribution p(m). 




Fig. 4. Layer=20. The histogram is simulation results (N — 10000, lOOOsamples). The line is a theoretical calcula- 
tion of the distribution p(m). 
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Fig. 6. Layer=100. The histogram is simulation results (N — 10000, lOOOsamples). The solid line is a theoretical 
calculation of the distribution p(m). 
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Fig. 7. p(m, a) (a = 0.2, 5 = 0.2, N = 200000, m° = 0.45, 1 = 100). 
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Fig. 8. p(cr). Integrating p(m, a) over m. 
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